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Abstract 

Since its introduction by P.L. Lions in his lectures and seminars at the College de France, see [S], 
and also the very helpful notes of Cardialaguet [J] on Lions’ lectures, the Master Equation has attracted 
a lot of interest, and various points of view have been expressed, see for example Carmona-Delarue [5], 
Bensoussan-Frehse-Yam [2] , Buckdahn-Li-Peng-Rainer [3] . There are several ways to introduce this type 
of equation; and in those mentioned works, they involve an argument which is a probability measure, 
while P.L. Lions has recently proposed the idea of working with the Hilbert space of square integrable 
random variables. Hence writing the equation is an issue; while another issue is its origin. In this article, 
we discuss all these various aspects, and our modeling argument relies heavily on a seminar at College de 
France delivered by P.L. Lions on November 14, 2014. 

1 INTRODUCTION 


Our objective in this paper is to present and discuss some of the main ideas and concepts related to the 
Master Equation, first introduced by P.L. Lions. Since the major source of information is only available on 
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video, our present presentation has also the objective to be a survey. The notes of Cardialaguet on Lions’s 
lectures are extremely valuable; however, they are not in the form as a survey. Besides, we cover some 
additional material, which is obtained more recently, including the seminar talk delivered by P.L. Lions on 
November 14, 2014, and our own contribution. The very exciting results of existence and uniqueness of the 
solution of the Master Equation mentioned by P.L. Lions in this seminar are unfortunately not covered here, 
since only the outline of their proofs were sketched in his talk. Instead, our present motivation is to provide 
a guidebook. A situation, which looks clear, is when one writes down a Bellman equation; it is the case 
for mean field type control problems, or mean field games which are equivalent to mean field type control 
problems. The Master Equation is then obtained by differentiating a Bellman equation, see [2]. One can 
have several writings but they all correspond to the same solution. Nevertheless, there are other cases, in 
which the Master Equation is not inherited from a Bellman equation. One important motivation in such 
cases is the fact that the Master Equation is a way to uncouple a forward-backward system of deterministic 
or stochastic P.D.E.s corresponding to mean field games or mean field type control problems. Another point 
of view is the fact that mean field games approximate differential games with a large number of players. The 
connection between the Master Equation and the large stochastic differential game also poses an interesting 
question. Einally, the linear quadratic model is quite useful, since one can often obtain explicit formulae, 
and so facilitates different comparisons. 


2 FUNCTIONALS OF PROBABILITY MEASURES 

2.1 DERIVATIVES 

In all problems connected with mean field theory, one naturally works with functionals of probability mea¬ 
sures on M”. An important issue is to define a convenient functional space. The natural space is the 
Wasserstein space . But in order to differentiate, this structure is not convenient since it is generally 
not a vector space. Two approaches are possible. Let m be a probability measure on M”. If it has a density, 
we shall use the same notation for the density, except that m = m(x), with x € M”. We then assume that 
m £ L‘^{W^). If we consider a functional F(m), we assume that it is dehned on and at the same time on 
We can use the concept of Gateaux differentiability on If F : —)■ M, then 


d 

dd 


F (m + Oifi) 


0=0 


I 


dF 

dm 




with ^ I— being in The second derivative (m) is a linear map from into 

such that 
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d dF 
dO dm 


(m + 9rh){^) 


e=o 


I 

JR’ 


92 F 

9?T7,2 


(m)(^, r])m(r])d7], 


where the function ^:^im){^,r]) is symmetric in {^,r]). 
We can state the second order Taylor’s formula 


f OF f f O^F 

F{m + fh) = F{m) + / —— {'m){^)rh{^)d^ + / ^ — K{m + Xfj,'m){^,ri)m{^)m{r])d^dridXd^ 

JR” dm Jq Jo JR” JR” 

( 2 . 1 ) 

The second possibility, introduced by P.L. Lions is to consider a probability space XI, A, P on which 
one can construct a random variable X on such that the probability law of X, denoted by Cx = 'm- 
We then write F{X) and consider that F is dehned on the Hilbert space H = L‘^{Xl,A,P',W^). We use 
the same notation F, to save notation. The fact that F{X) = F[m) = F{Cx) means that the functional 
F{X) depends of X only through its probability. This implies specihc aspects, as we shall see. An obvious 
advantage of this approach compared to the previous one is that it is more general, since the probability 
measure does not need to have a density. Besides, because we use random variables, we keep the fact 
that m is a probability, and is not a general function in We can then use the concept of Gateaux 

differentiability on Ti . We denote the scalar product in P by {X,Y). Of course {X,Y) = KX.Y, but it is 
good to consider P as a general Hilbert space. We then have 

^F{X + eY)\0=o = {DF{X),Y) ( 2 . 2 ) 

and DF{X)& P. We then proceed to define the 2nd derivative 

^{DF{X + OY), Z)\e=o = {D^F{X)Y, Z) = D^FiX){Y, Z) 
in which D‘^F{X)£ C{P;P) is identified to a symmetric bilinear form on P. We can then write as for 

m 


1 /■! 


F{X + Y) = F{X) + {DF{X),Y) + 



XD^F{X + Xfj.Y){Y,Y) 


(2.3) 


0 JO 


0 F (tti) 

The first question is what is the relation between DF(X) and —- (x) ? 

dm 


To answer this question we consider a random variable Y and assume that the pair X, Y has a joint 
probability density Tr{x,y). By dehnition the marginal of X has the density m{x). We dehne the stochastic 
process ^(t) = X + tY. It is easy to check that for t > 0 , ^{t) has a probability density m{x, t) given by the 
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formula 


m{x, t) 



ty,y)dy 


so 


dm , , 

— (x,f) = -dlVa; 



ty,y)ydy 


But then 


d_ 

dt 


/ Fi F)ty) 

= -J ^^{m{t)){x)div^ j 7r{x 


ty,y) ydydx 


therefore 


_d 

dt 


/ dF f 

— (m)(x)div 3 ; / Tr{x, y) ydydx 

/ dF f 

-Dx^(m.)(x).(/ 7r{x,y)ydy)dx 


From the definition of 7r(x, y) this can be written as 


rl f)F 

-F{mmt=o = nD.^{m){X).Y] 


On the other hand, since F{m{t)) = F{^{t)) we have also 


|F(m(t))|i=o = j^Fm)\t=o = nDF{X).Y] 


and since Y is arbitrary, we have obtained the formula 


DF{X) = D^ — {m){X) (2.4) 

This proves a result which can be obtained in a more general set-up. The derivative DF{X) can be 
computed through a function of m and x, denoted (using the notation of ^)dmF{m){x), with the formula 
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DF{X) = dmF{m){X) 


(2.5) 


Of course from the previous calculation, we can state that when the probability has a density denoted 
by m, then 


dmF{m){x) = D^ — {m){x) (2.6) 

We can apply this result to the functional F{X) = EF(X, m), which means 

F{X) = / F{x,m)m{x)dx (2.7) 

We see easily , by formula (12.411 that 

DF{X) = D^F{X, m)+D, [ ^F(^, m){X)m{Od^ (2.8) 

jR" am 

We will have a more complex situation to handle. We consider a function V(x,m) :M”' x L^(M”') W^. 

Next we set U{X) = V{X,Cx) ■ Then U maps % into itself. 

We want to compute its gradient DU{X) which is a map from % into C{T-L]TF). If Z £ T-L then for any 
W in we have 


{DU{X)Z,W) = {Dx{U{X),W),Z) = 


(2.9) 


dV, 


= ^J2^3-^(^^^x)Wi + {DxEj2ViiY,Cx)Wi,Z) 


in which Y,W is a copy of X, W independent of X, Z. This implies that the dependence in X in the 
second term is only trough Cx- We then have 


{DxEY,V.(Y,Cx)W„Z)=J2BxzZ,-YY^Y,V,(Y,CxKX)Wi] 

i j ^ i 

with the notation ^x){X)Wi = Cx){x)Wi\x=x-, namely we take the expectation 

with respect to the pair Y, W with x deterministic, and we replace in the result x by the random variable X. 
When this is done, we take the expectation in the pair X, Z, the only random variables remaining. We can 
then rename Y, W into X, W and X, Z into Y, Z in which Z is a copy of Z, such that T, Z is independent 
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of {X^W). We get ( with yj replacing accordingly Xj for the coherence) 


[Dx^Y.^i{Y,Cx)Wi,Z) = Y,^yzZj — — [ExwY.^i{X,Cx){Y)Wi] = 

i j 'i' 


= E.vir E »'i E Cx)(Y) = 

i j yj 


= WEEv.^,^^r(A,£,)(F)) = 

= {W,EyxD„Xv(x,CxmZ) 


On the other hand 


j ^ 


therefore we conclude that 


Dl({X)Z = D,V{X, Cx)Z + Eyz By-£-V(X, Cx){Y) Z (2.10) 

Suppose we apply this formula to V{x, m) = DxU{x, m) where U{x, m) is scalar, thenZ^(X) = DxU{X, Cx) 
therefore (j2.1U|) implies 


DU{X)Z = DlU{X,Cx)Z + DyMYzDy^U{X,Cx){Y).Z (2.11) 

Q 

In the case Uix^m) = —F(m)(x), ihenUiX) = DFiCx))^ hence (12.lip implies 

dm 

D^F{Cx)Z = dIXf{Cx){X)Z + Ey2D^Dy-XF{Cx){X, Y)Z (2.12) 

and in the second term Y, Z is an independent copy of (X, Z). 

2.2 APPLICATIONS 

Suppose we apply (I2.12|l to Z = il(A)r, in wich T is independent of X and satisfies EF = 0, then we can 
take Z = il(y)r , in which P is a copy of Pwith P independent of Y and the pair (T, P) is independent of 
X, P. In that case, the 2nd term vanishes and it remains 


6 









2 d 


D^F{Cx)B{X)T = Di—F{Cx){X)B{X)T 


(2.13) 


We apply this formula with B{x) = cf{x) and T a vector of n independent variables, also independent of 
X and with mean 0 and variance 1. We have 


{D^F{Cx)a{X)T,a{X)T)=¥.Dl^F{Cx){X)a{X)T.a{X)T = 


= E 


dm 

Dl 


A 

dm 


F{m){x)a{x)T .a{x)Tm{x)dx 


from which it follows easily 


{D^F{£x)aiX)T,a{X)T) 



d 

tr {a{x)D‘^— — F{m){x))m{x)dx 


(2.14) 


We next derive another formula. Denote by e^, /c = 1, • • • , n the vector coordinates in Consider 


.2 d 


D^F{Cx)ek = Dl-^F{Cx){X)ek + D, 


D 


= Dt 


A 

dm 


A_ 

y Qjy^2 
2 


F{Cx){X)ek - D, -^F{m){X,y) 


F{m){X, y)ekm{y)dy 
dm{y) 


dyk 


-dy 


Therefore 


{D‘^F{Cx)ek,ek) = f ■^-^F{m){x)m{x)dx + [ ■^-^F{m){x,y) 


dm{x) dm{y) 

dxk dyk 


dy 


hence 


n f d r r d^ 

V(D^F(£x)efc,efc) = / —F{m){x)Am{x)dx+ / --^F{m){x,y)Dm{x).Dm{y)dxdy (2.15) 

Jr" dm J^n J^n dm 


Using (|2.14p this is also written as 


j2{D^FiCx)ek,ek) - {D^F{Cx)r,T) = [ [ 

JR" JR^ 


A_ 

dm'^ 


F{m){x, y)Dm{x).Dm{y)dxdy 


(2.16) 


We can recover formulas (|2.15p . p2.16D by an approach relying on stochastic differential equations (SDE) 
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We first consider the solution of the SDE 


dx = a{x)dw{t) (2-17) 

x(0) = X 

and Cx = The Wiener process w{t) is a standard Wiener process in M” which is independent of 
X. Define the matrix a{x) = cr{x)a*{x). The random variable x{t) has a density m{x,t) solution of the 
Fokker-Planck equation 


dm 



dxidxj 


{aij{x)m) = 0 


m{x, 0) = m(x) 


Therefore 


d f dF , , , w N dm , , , 

—F{m{t)) = j —{m{t)){x) — {x,t)dx = 


_1 ^ /■ , dF 

2 J ^ dxidxj dm 

i,j=i •' * J 


{m{t)){x)m{x, t)dx 


from which it follows 


J 1 /* riJ^ 

—F(m(f))|t=o = 2]^^ {a{x)Dl — {m){x))mix)dx 

but because of (12.bp we can state immediately the formula 


,2dF 


Dx-^{m){x) = D^dmF{m){x) 
dm 


(2.18) 


therefore 




tr {a{x)DxdmF(m){x))m{x)dx 


(2.19) 


On the other hand, for t small we can approximate the process x{t) by 




x{t) ^ X + Vta{X)N 


where is a standard gaussian independent of X with values in M”'. We can then write 


F{m{t)) ~ F{X + Vta{X)N) 


and from formula 


F{X + Via{X)N) = F{X) + {DF{X), Via{X)N)+ 

+ [ [ XD^F{X + XnVia{X)N){Via{X)N,Via{X)N) 

Jo Jo 

The 2nd term on the right hand side is 0, since N and X are independent. It follows easily 


F{X + y/ta{X)N) ~ F{X) + ^D‘^F{X){a{X)N,cr{X)N) 


which imediately leads to 


/ f) /? 

tr {aix)Di— — (m)(x))m(x)dx = 
dm 

Consider now the following model 


( 2 . 20 ) 


x{t) = X + /3b{t) 

in which b{t) is a standard Wiener process in M”, and /3 is a constant. We denote by the a—algebra 
generated by b{s),s < t. We consider then the conditional probability of x{t) given B^, so which we 

denote again by m(t) = m{x,t). We have obviously 

m{x, t) = m{x — fJb(t)) 

so the function m{x, t) is random and B*^ measurable, as of course expected from the definition. This time 
m{x, t) satisfies a stochastic partial differential equation SPDE, 
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/32 

dtm ——Am dt + fdDm.db = 0 
m{x, 0) = m{x) 


From mi we can derive the following Ito’s formula 


dF{m{t)) = j ^-{m{t)){i)[^Am{i,t)dt - PDm{i,t).dh{t)]di J 
f f d'^F 


+ 


dm^ 


r])Dm{^, t).Dm{r], t)d^dr]dt 


( 2 . 21 ) 


from which we get easily 


J fF f 

-IEF(m(t))|,=o = ^J ^(m)(0Am(0dC+ 


+ 




92 F 

5m 2 


(m)(^, Tj) Dm{^) .Dm{r})d^dr] 


( 2 . 22 ) 


On the other hand F{m{t)) = F{X + /36(t)).Using formula (|2.3I) we have 


F{X + Ph{t)) = F{X) + PiDF{X), b{t))+ 

+P^ f [\d^F{X + XixPb{t)){b{t),bit)) 

Jo Jo 

from which it follows easily that 

d 52 ” 

-IEF(m(t))|t=o = Y E D^nX){ek,ek) (2.23) 

k=l 

in which are the coordinates vectors in M”. Comparing with (I2.22p we have shown the relation 

^D‘^F{X){ek,ek) = / —{m){i)Am{p)di+ / / -—{m){^,r])Dm{^).Dm{r])d^dr] (2.24) 

fc=i ■' 

Using (|2.2()p we get also 


10 









(2.25) 


r r F ^ 

J J ^i'^)i^^v)Dm{0.Dm{ri)dCd7] = Y^^D^F{X)iek,ek) - D^F{X){N,N) 

Remarkl. lfF{Cx) = F(x)m(x)dx. then it follows from (|2.16p that 'F‘^-i(D‘^F( FY)ei..ei.) — (D‘^F( Fv)T. ri 

0 . 

2.3 APPROACH OF BUCKDAHN et al. 

Remembering the definition of dmF{m){x),see (|2.5p . it is natural to extend it to the 2nd derivative, con¬ 
sidering X fixed and thus dmF{m){x) as a function of m. This is the approach proposed by [3]. Note that 
dmF{m){x) is a vector in M”. So consider the component dm,iF{m){x). We associate to it the function of 
the random variable X , denoted dm,iF{X){x). We can define its derivative D dm,iF{X){x) as in (|2.2I) . if of 
course, the function has a derivative. But then there exists a new function dm{dm,iF{'m){x)){y) such that 

dra{dm,iF{m){x)){X) = Ddm,iF{X){x) 

We then write 


dm,j{dm,iF{m){x)){y) = d^^ijF{m){x,y) 
This is a new concept. We can from the definitions establish the relation 

di,ijFim)ix,y) = Dy^-^{D^.^^^{x)){y) 

We can check that 


therefore 


dm 


D 


Xi 


dF{m) ^ ^ ^ d‘^F{m) 

{x)){y) = D^^—^^{x,y) 


dm 


d‘L,ijF{m){x,y) = Dy.D^. 


d‘^F{m) 

dm^ 


{x,y) 


So we may write 


dmF{m){x,y) = DyD, 


d^F{m) 

dm^ 


{x,y) = D^D, 


d‘^F{m) 
' dm^ 


{x,y) 


and going back to (|2.12p we obtain 


(2.26) 


(2.27) 


(2.28) 


(2.29) 


11 













D‘^F{Cx)Z = Dl^F{Cx){X)Z + Ey^idiF{m){X,Y)Z) 

which gives the relation between the second derivative introduced by Buckdahn et al. and the second 
derivative in the Hilbert space T-L. 

In this set up, Taylor’s formula is more complex. Consider two probabilities mo and m to which corre¬ 
spond random variables Xq and X. We use (|2.3p with the notation of ([3]). 

We have 


F{X) - F{Xo) = {DF{Xo),X- Xq) + [' f \D^F{Xo + A^(X - Xo))(X - Xo,X - Xo)dXd^i 

Jo Jo 

Now from formula (I2.12|) we can write 

{D^F{Cx)Z, Z) = EDl^F{Cx){X)Z.Z + Exz Z.EyzD^Dy^F{Cx){X, Y)Z (2.30) 
Using this formula and notattion ([3]) we obtain 


F{m) - F{mo) = (9™F(mo)(Xo), X - Xo)+ 


(2.31) 


+ [' C XExxoiX - Xo).Eyyo dlF{Cx,+x^,ix-Xo)){Xo + XfM{X - Xo),Yo + X^,{Y - Yo)){Y - Yo)dXd^l+ 
Jo Jo 


+ r r XED,^mF{/:xo+x^.(x-Xo})iXo + A(X - Xo)){X - Xo).{X - Xo)dXdfi 
Jo Jo 


which we write as follows 


F{m) - F{mo) =EdmF{mo){Xo).{X - Xo)+ 


(2.32) 


+ ^Exxo{X - Xo).Eyyo dlF{mo){Xo, Yo){Y - Tq) + ^ED^dmF{mo){XoKX - Xo).{X - Xq) + R{mo; m) 

Buckdahn et al. [3] have shown that R{mo', m)—)■ 0 with the order of E|X — with appropriate assump¬ 
tions on the second derivatives. Formulas (j2.31l] and (j2.32D are substantially different from (12.11) or (12.3p . 
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This is because of the two variables m and x instead of a single one m or X. 


3 DYNAMIC PROGRAMMING 

3.1 MEAN-FIELD TYPE CONTROL 

We consider a probability space -P on which are defined various Wiener processes. We define first a 
standard Wiener process w{t) in M"'. The classical mean field type control problem is the following; Let 
v{x) be a feedback with values in the corresponding state equation is the Mac Kean-Vlasov equation 


dx = g{x, u(x))(it -|- a{x)dw (3-1) 

x(0) = xo 


in which m^(,)(x,t) is the probability density of the state x{t). The initial value xq is a random variable 
independent of the Wiener process w{.). This density is well defined provided we assume the invertibility of 
a(x) = a{x)a*{x) . We define the 2nd order differential operator 


and its adjoint 


Aip{x) 


1 

2 


^aij{x) 


d‘^ip{x) 

dxidxj 


Next define 


A*(f{x) 


1 d‘^{aij{x)ip{x)) 

2 “ dxidxj 


J(x(.))=E[/' /(x(t),m„(.)(t),u(x(t)))(it-k/i(x(T),m^(.)(r))] (3.2) 

Jo 

The mean field type control problem is to minimize J{v{.)). Note that the feedback v{x) can also depend 
on m. It will be indeed the case for the optimal one. The problem can be easily transformed into a control 
problem for a state, which is the probability density ). It is the solution of the Fokker-Planck equation 


dt 


+ + div ( 5 (x,m^(.),x(x))m„(.)(x)) 


m(x, 0) = mo(x) 


(3.3) 
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in which mo{x) is the probability density of the initial value xq. The objective functional J{v{.)) can be 
written as 


/ f{x,my^,){t),v{x))my^,){x,t)dxdt+ /i(x,m„(.)(r))m^(.)(a:,T)dx (3.4) 

Jo Jr^ Jr^ 

We next use the traditional invariant embedding approach. Define a family of problems, indexed by 
initial conditions m, t 


ds 


+ yl*m^(.) + div ( 5 -(x,m^(.),u(x))m„(,)(x)) 


m{x, t) = m{x) 


(3.5) 


/ fix,my(,){s),v{x))my(,){x,s)dxds+ /i(x,m^(.)(r))m^(.)(x,r)(ix (3.6) 

Jt v/R" v/R" 


and we set 


V{m,t) = inf JmA'^i)) 

vi.) 


(3.7) 


Using the optimality principle one obtains 


dV 

'dt 


dV{m) 

dm 


{^)A*m{Od^+ 


(3.8) 


+ mi( f f{tm,v{^))m{^)dC - [ ^^^(Odiv (ff(^, m, u(C))m(0)dc) = 0 
^ \Jr^ Jr^ dm ) 


and 


V{m,T)= / h{x,m)m{x)dx 


(3.9) 


The infimum is attained by minimizing inside the integral. Therefore, if we define the Hamiltonian 
H{x, m, q) , with g G M" by 


H{x, m, q) = inf(/(x, m, v) + q.g{x, m, v)) 


(3.10) 
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we can write as follows 


[ A^^{0m{0dC+ [ H{tm,D^^^{O)m{OdC = 0 (3.11) 

Jm" Jr" 

So p.llD . (I3.9|) is the Bellman equation for the mean-field type control problem (I3.3D . (13.40 . with deriva¬ 
tives with respect to densities. 

The next step is to write the corresponding equation for V {X, t) with the argument X in T-L . We use 
(|2.4|) and (j2.140 . 

Then (|3.1ip becomes 


^ + ^D^V{X){a{X)T,a{X)T) +EH{X, Cx, DV{X)) = 0 (3.12) 

V{X,T)=Eh{X,Cx) 

Example 2. If we take cr = 0, g{x,m,v) = v, f{x,m,v) = ^|np,we get the Eikonal equation 

f-1||W{V)||^ = 0 

3.2 STOCHASTIC MEAN FIELD TYPE CONTROL 
3.2.1 PRELIMINARIES 

If we look at the formulation (j3.3p . (13.40 of the mean field type control problem, it is a deterministic problem, 
although at the origin it was a stochastic one, see (j3.ip . (13.20 . We now consider a stochastic version of (j3.3p . 
(13.40 or a doubly stochastic version of (13.10 . (j3.20 . Let us begin with this one. Assume there is a 2nd 
standard Wiener process h{t) with values in b{t) and w{t) are independent and independent of xq. We 
set B^=cr{b{s), s < t) and J^=a{xo,b{s),w{s), s < t) . The control v{x,t) at time t is a feedback, but not 
deterministic. It is a random variable, measurable. We consider the stochastic Me Kean-Vlasov equation 


dx = g{x, {t),v{x, t))dt + a{x)dw + j3db{t) (3.13) 

x(0) = xo 

in which m^(,)(t) represents the conditional probability density of x{t), given the cr—algebra B^ . We want 
to minimize the objective functional 
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(3.14) 


Jiv{.)) =IE[/' f{x{t),m^(^,){t),v{x{t),t))dt + h{x{T),m^(^,){T))] 
Jo 

3.2.2 CONDITIONAL PROBABILITY 

If we define 


y{t) = x{t) - I3b{t) 


then the process y{t) satisfies the equation 


dy = g{y{t) + /36(t), m„(.) {t),v{y{t) + I3b{t),t))dt + a{y{t) + /3b{t))dw (3.15) 

y(o) = xo 

If we hx b{s), s <t then the conditional probability of y{t) is simply the probability density arising from the 
Wiener process w{t),in view of the independence of w{t) and b{t). It is the function p{y,t) solution of 


dp 


1 

2 


ij 


d^p 

dyidyj 


)+ 


+div (^g{y{t) + /3b{t),m^(^^){t),v{y{t) + /3b{t),t))p^ =0 
p{y,0) = mo{y) 

The conditional probability density of x{t) given is the function 


m{x, t) = p{x — f3b{t),t) 

hence 

dtm = (^ + ^/3‘^Ap){x — /db{t),t)dt — j3Dp{x — /db{t),t)db{t) 

which yields 

dtm + {A*m — + div( 5 ((x, m{t),v{x, t))m))dt + l3Dm.db{t) = 0 (3.16) 
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m(x, 0) = mo{x) 


and the objective functional can be written as 


J{v{.))='E[f f f{x,m{t),v{x,t))m{x,t)dxdt+ f h{x,m{T))m{x,T)dx] (3-17) 

Jo v/r" 

The problem becomes a stochastic control problem for a distributed parameter system. Using invariant 
embedding, we consider the family of problems indexed by m, t 


dsTH + {A*m — + div( 5 (x, m{s),v{x, s))m))ds + l3Dm.db{s) = 0 


(3.18) 


and 


Set 


m{x, t) = m{x) 


Jr 


=¥.[ f f f{x,m{s),v{x,s))m{x,s)dxdt+ f h{x,m{T))m{x,T)dx\ (3.19) 

Jt iR" Jmx 


V{m,t) = inf JmA'^i-)) 

v(.) 

then V {m, t) satishes the Dynamic Programming equation 


(3.20) 


2 2 Jffin Jffin dm^ 


dt 


I 

JR'' 


dm 


(3.21) 


+ mf( f f{^,m,v{0)m{^)dC - [ (Odiv (g(^, m, v{^))m{^))dC 

^ VjRn JiRn dm 

V{m,T)= / h{x,m)m{x)dx 
Jr" 


= 0 


which we rewrite as follows 


Ir ~ ^^dm ^ ^J^2’^\ c,r])Dm{C)Dm{r])dCdr]+ (3.22) 
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+ 




Let us write the equation for the corresponding function V{X,t). We use now (I2.1hp and we obtain, after 
cancellation of the term F) 


—+ -L»V(X)(c7(X)r,c7(X)r) + -/ 32 ^L»V(X)(efc,efc)+EF(X,£x,W(X)) = 0 (3.23) 

k=l 


V{X,T)=Eh{X,Cx) 


This write up is more condensed than (I3.2ip . 


4 FIRST TYPE OF MASTER EQUATION 

4.1 THE PROBLEM OF BUCKDAHN et al. 

In their paper [3], the authors consider the following problem. It is not a control problem. As in (13.5D , let 
m{x, s),s >t solution of 


—-h A*m + div {g(x, m(s))m(x, s)) (4.1) 

os 

m{x, t) = m{x) 

and we consider the associated diffusion 

dx = g{x{s),m{s))ds + a{x{s))dw{s) (4-2) 

x{t) = X 

Of course, if the initial condition x{t) were replaced with a random variable independant of the Wiener 
process, with probability density m{x), then m{x,s) would be simply the probability density of x(s).Since 
it is not the case, the pair ( 14.ip . (14. 2 p is a system. The authors are interested in the functional 

U{x, m, t) = Eh{x{T),m{T)) (4.3) 

Note indeed that the solution of (|4.ip can be written as mm,t{x, s) and the solution of (14.2D as Xm,x,t{s), 
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to emphasize the dependence in the initial conditions, and this justifies the notation (Id.lll) . We will write 
the formulation of the equation satisfied by U with the 3 approaches explained above. 


4.2 VARIOUS FORMULATIONS 

We begin by considering that m is evolving in We can see formally , using the Markov property of 

the pair m{s),x{s) that 




dU{x, m, t) 
dm 




(4.4) 


+DxU.g{x, m) — AU(x, m,t) = 0 


U(x, m, T) = h{x, m) 


We write it as 




A, 


dU{x, m, t) 
dm 


{C)m{^)d^+ [ 

Jw 


D, 


dU (x, m, t) 
dm 


{^).g{i,m)m{i)di+ 


+DxU.g{x^ m) — A^U{x, m,t) = 0 


(4.5) 


Because of the two variables x, ^ it is important to clarify on which variables the differential operators 
act. For that purpose we have used the notation D^, and Ax.,A^. Using formulas (|2.6I) and (I2.18p we have 


^ + ^^{a{0D^dmU{x,m,t){0)M0d^+ [ dmU{x,m,t){^).g{^,m)m{^)d^+ (4.6) 

^ Jr** Jr** 

+DxU.g{x, m) — AxU(x, m,t) = 0 

U(x, m, T) = h{x, m) 

This is the equation studied rigorously by Buckdahn et al [3]. The authors notice that there is no second 
derivative in m,although there are second derivatives in x. This is explained by the fact that the equation 
in m is deterministic, whereas the equation in x is random. Let us now write the equation in the space %. 
We consider U{x,X,t) = U{x,Cx,t). 

From (j2.4p and (j2.2()l] we obtain 


^ + \dIU{x, X, t){a{X)r, c 7 (A)r) + (DxUix, X, t),g{X, Lx , t))+ 


(4.7) 
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+DxU{x,X,t).g{x,Cx,t) - A^U{x,X,t) = 0 


U{x,X,T) = h{x,Cx) 

In this set up, we obtain a more symmetric formulation. There is no mixed derivatives and there is a 
second order derivative in X. 

Remark 3. From the methods described above, it is possible to extend the equation (1121) to nonlinear 
situations. 


5 OBTAINING THE MASTER EQUATION BY DIFFERENTIATING 
BELLMAN EQUATION 

5.1 THE MEAN FIELD TYPE CONTROL CASE. 

We first go back to (I3.10|) and assume that the minimum is attained in a unique point which we denote by 
v{x,m,q). We then set 


G{x, m, q) = g{x, m, v{x, m, q)) 


(5.1) 


We have also 


G{x, m, q) = DqH (x, m, q) 


We then consider the value function V (m, t) defined in (|3.2Up and set 


TT, ^ dV(m,t), . 
U{x,m,t) = 


we write (13.lip as 


(5.2) 


(5.3) 


dV 


f AU{^,m,t)m{^)d^ + f = 0 

Ir" Ir" 

We next differentiate (15.4p in m . We note that 


A 

dm 


f DU{^,m,t))m{^)d^]{x) = H{x,m,DU{x))+ 


(5.4) 


+ [ ■^H{^,m,DU{0)ix)m{^)dC+ [ G{^,m, DU{^))m{^)D^^U{^,m,t){x)d^ 
J^n am J^n dm 


20 









Hence, using 


—U{i,m,t){x) = —U{x,m,t){^) 
am am 


d‘^V{m,t) 

dm? 


(3^,0 


we obtain 


-^ + AU+ ! ^[/(x,m,t)(e)(A*m(0+div(G(e,m,D[/(0)m(0K 

at J-^n dm 

r ^ 

H{x,m,DU{x)) + / —H{^,m,DU{^)){x)m{^)d^ 

Jk" am 

f d 

U{x,m,T) = h{x,m) + / —— h{^,m){x)m{^)d^ 


We write this equation as 


(5.5) 


(5.6) 


Ot T /* C) C) 

+ H 3 ,C/ + {A^ — U{x,m,t){C)-D^ — U{x,m,t){C)-G{i,m,DU{i)))m{i)di= (5.7) 

II(x,m,Dl7(x)) + / —ff(^,m,D[/(^))(x)m(^)d^ 

Jj^n dm 


f 9 

U{x,m,T) = h{x,m) + / —— h{^,m){x)m(^)d^ 

J^n dm 

5.2 MASTER EQUATION IN H 

We then introduce ^^(A, t) = DV{Cx,t)- So U maps 7i x [0, T] into Ti. To simplify a little we take cr(a:) = al, 
then Bellman equation (15.121) reads 


^ + ^D^ViX)iT,r)+EHiX,Cx,U{X)) = 0 

V{X,T)=Eh{X,Cx) 


(5.8) 


We proceed with the formal differentiation of (j5.8|) . First DU{X) : "H x [0,T] —>■ So DU{X) is 

a random n x n matrix . We obtain 


DEH{X,CxM{X)) = D,H{X,Cx,U{X)) + {DU{X))*G{X,CxM{X))+ (5.9) 
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dH 


+W.Y,uiY)D.^{Y,CxM{Y)){X) 


in which Y,V({Y) is an independent copy of X,U{X). But Dhl{X) = D‘^V{Cx,t) is self-adjoint , so we 


obtain that V({X,t) satisfies the equation 

+ ^D^U{r,r) + DU{X)G{X,£x,U{X))+ 


(5.10) 


dH, 


D,HiX,Cx,H{X)) + IEy,w(y) D, — {Y,Cx,UiY)){X)=0 


U{X,T) 


3h 

D^h{X,Cx) + ^yD, — {Y,Cx){X) 


We also can obtain this equation from (|5.7I) . For ct(x) = al, this equation writes 


y -h y J A^—U{x,m,t){^)m{^)di + J D^—U{x,m,t){C)-G{C,m,DU{^)))m{^)dC+ 

(5.11) 


+H{x, m, D^U{x)) + [ m, D^U{0){x)m{^)dC = 0 

j-^n om 


We differentiate in x,so we get 


d 

-D,U+—A^D^U+—D, 


[ X^-^U{x,m,t){^)m{^)d^+Da; [ 

Jr" dm Jr7 


(5.12) 


C 3 

+DxH{x,m, DxU{x)) + DlU{x,m,t)G{x,m, D^U) +D^ —H{^,m,D^U{()){x)m{^)d^ = 0 

jRn dm 

We recall that DxU{X, m,t) = DV (X, t) = U{X, t). Now from formula (I2.14p we have, skipping the argument 
t 


D‘^V{X)(T,T)= [ AxUix,m)m{x)dx 
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If we differentiate in X the left hand side, we get D^ly/(X)(r, P). But the right hand side is a functional of m. 
So we can obtain the same result , by first differentiating the right hand side in m,then the result in x,and 
finally replacing x by X. Calling F{m) the right hand side, we first have ( using the symmetry property) 


dF{m) 

dm 


(x) = A.xU{x,m) + 


= AxU{x,m) + 


(^, m){x)m{^)d^ 


Hence 


^ dF{m) 

x 

am 


(x) 


AxDxU (x, m) F Dx 



A^^U{x,m){i)m{i)di 

am 


Replacing x by X, we conclude 


r o 

D^U{X){T, P) = AxDxU{X, Cx) + Dx / Ag—[/(X, 


(5.13) 


We next have 


DxH{X,m,DxU{X)) = DxH{X, CxM{X)) 


(5.14) 


and 


r n lATj 

Dxj^^ -^H{i,m,D^U{i)){X)m{i)di = ¥.Y,uiY)Dx^{YXxM{Y)){X) (5.15) 

Finally it remains to check that 


D, 


I 

JR'' 


D^-^U{X, m, t)(O.G(C m, + DIU{X, m, t)G{X, m, DxU{X)) = (5.16) 


DU{X,t)G{X,Cx,G{X)) 


Using formula (|2.12p with Z = G{X, Cx,G{X)) we obtain 


DU{X,t)G{X,CxM{X)) = DlU{X,Cx,t)G{XXxM{X))+ 

r o 

+ / DxD^ — UiX,Cx,tmGi^,Cx,D^UiO)m{Od^ 

JR" am 
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which is exactly (I5.1fip . 


5.3 INTERPRETATION OF THE MASTER EQUATION 

We can interpret U{x,m,t) as uncoupling the system of HJB-FP equations of the mean-field type control 
problem, see [2]. We recall briefly the idea. 

The probability density, corresponding to the optimal feedback control is given by 


——h A*m + div (G(x, m, DU)m(x)) = 0 
at 

m{x, 0 ) = mo(x) 

Define u{x,t) = U{x,m{t),t), then clearly, from (I5.5I) . (|5.6I) we obtain 

(5.17) 

(5.18) 


du C d 

— + Au = H(x,m, Du(x)) + / 7 :— Du(0)(x)m(^)d^ 
at jRn dm 

f d 

u{x,T) = h(x,m) + / —— h(^,m){x)m(^)d^ 


which together with the F-P equation 


dnij 

——h A*m + div (G(x, m, Du)m(x)) = 0 
at 


m{x, 0 ) = mo(x) 


(5.19) 

(5.20) 


form the system of coupled HJB- FP equations of the classical mean held type control problem, see [T]. 

5.4 STOCHASTIC MEAN FIELD TYPE CONTROL 

We go back to (|3.22p . We dehne again U{x,m,t) = write 

U{x,m,T) = h{x,m) + [ — 77 ^-^(x)m(Qd^ (5-21) 

jjjn dm 

and from (13.221) we obtain 
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dV 


[ - ^(3‘^AU){^, m, t)d^+ 


( 5 . 22 ) 


{ri)Dm{^)Dm{ri)d^dri + f H{^,m, D^U)m{^)dx = 0 
2 Jm" Jm" ow- jRn 

We then differentiate this equation in m, to get an equation for U. We note that 


A 

dm 


f f ^^^i’'^’^\'n)Dm{^)Dm{'n)d^d7]] 
Jr” Jr” am / 

{x)= [ [ 

JR” JR' 


—2div ( / 

\Jr” 


r])Dm{^)Dm{r])d^dr]- 

iv)Dm{r])dr]^ 


and 


A 

dm 


H{^,m,DU)m{^)dx\=H{x,m,DU{x))+ f D^U{^)){x)m{^)d^+ 

/ JR” dm 

+ f ■^{D^U{^,m,t)){x)G{(,m, DU{^))m{()d^ 

J-^n am 


Next 


- [ -^U (C, m, t)(x) div(G(C, m, 
jR" am 


DU{i))m{i)di = [ m, t)(x))G(C, m, DU{C))m{m = 

JR” 

/* 

DU{^))m{())d^ = - —U{x,m,t){0 div{G{^,m, DU{^))m{^))dC 

JR” am 


Collecting results, we obtain the Master equation 




(5.23) 


+ f {M-^U{x,m,t){i)-D^^U{x,m,t){C)-G{i,m,DU{i)))m{i)di-]^P^ 


A[/ rn, t){C)A^m{()dC 
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/ / 


d‘^U{x ,m,t) fl2j- f f dU{x,m,t) 


dm? 


-(^, rf)Dm{^)Dm{r])d^dij + /3^div 


5 m 


{^DmiOdi 


= H{x 


,m,DU{x))+ f -^H{i,m,DU{i)){x)m{i)di 

JR" 


U{x,m,T) = h{x,m) + f ——S^-^{x)m{^)d^ 

JR" 


(5.24) 


We next write the Master equation in the space Ti , for U{X) = DV{X) = DxU{X, Cx)- Recall that we 
take a{x) = /. From (j3.23p and (|5.1Up we get 


Q 2 1 n 

-U + ^D^UiT, F) + Y. D^id{X){ek, ek) + DU{X)G{X, CxM{X))+ 


(5.25) 


k=l 


f) M 

DxH{X,Cx,U{X)) + EY,uiY)Dx^iY, Cx,U{Y)){X) = 0 


f)h 

U{X,T) = Dxh{X,Cx) + EyDx^{Y,Cx){X) 

om 

If we use the Master equation p5.23D . p5.24D , then from the calculations done in the case /3 = 0, what 
remains to be proven is 


^ r p) 

Y D‘^U{X){ek, ek) = D^A^UiX, Cx) + / ^U{X, m, t){0AM0dC+ (5-26) 

k=l JR" 


+A 


/ / 

JR" JR’ 


d U{X, m, t) n)Dm{p)Dm{ri)didri - 2Dxd\Y ( f {^^)Dm{i)di 

VJr" dm 


dm"^ 


From formula p2.15p we can write 


n (' d f f C? 

Y{Y>'^Y{Cx)ek,ek) = / ■ 7 ^V{m){x)Am{x)dx + / / -—^V{m){x,y)Dm{x).Dm{y)dxdy 

JR" JR" JR" dm 


which can be rewritten as 
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(5.27) 


n p 

Y,DUiX)iek,ek)= / 

k=i 


U{x, m)Am{x)dx + 


[ [ 

Jr" Jr" 


X, m){y)Dm{x).Dm{y)dxdy 


= f AU{x,m)m{x)dx + f f — — U{x,m){y)Dm{x).Dm{y)dxdy 
Jr" Jw^ Jw^ 

If we differentiate in X, the left hand side gives D'^^{X){ek, e^), which we want to compute. The right 
hand side is a functional of m. So we first take the Gateaux differential in m, which is 


AU{x, m) + 


A^-^U{i,m){x)m{i)di+ 

am 


-\- [ [ ^ ^^'^'^\ ^,r])Dm{^)Dm{r])d^dr] - 2div ( f l^ri)Dm{r])dr]\ 

Jr‘^ Jr‘^ \Jr^ / 

which is also 


f d 

A^U{x,m)+ —U{x,m){C)Am{C)dC+ 
jRn am 

Jr" jr" vyR" / 

We then have to take the gradient in x, and replace x by X. We obtain immediately the relation (15.261) . 

5.5 SYSTEM OF HJB-FP EQUATIONS 

As in the deterministic case /3 = 0, see section 15.31 we can derive from the Master equation a system of 
coupled stochastic HJB-FP equations. They reduce to the deterministic system (|5.17D to (|5.20p when /3 = 0. 
Consider the conditional probability density corresponding to the optimal feedback v(x, m, DU(x, m, t)), see 
dsn), solution of 


dtm + {A*m — ^/3^Am + div(G(a:, m, DU)m))dt + j3Dm.db{t) = 0 


(5.28) 


m(x, 0) = mo{x) 

and set u{x,t) = U{x,m{t),t), we obtain 
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(5.29) 


—dtu + {Au — -f3‘^Au)dt + /3^div ( [ —— dt = 

2 VJ]Rn dm / 

H{x,m,Du{x))+ [ ■^H{^,m,Du{^)){x)m{^)d^ + P f {^)Dm{^)d^db{t) 

Jr" am jjjn am 


f d) 

tt(x, T) =/i(x, m) + / —— h{^,m){x)m{p)d^ 

Jr" 


(5.30) 


Setting 



dU{x, m, t) 
dm 


{i)Dm{0di 


we can rewrite (|5.28p . (|5.29p as follows 


—(9tn + (^n — ■^/3^An)dt + /3^div B{x, t)dt = (5.31) 

/» o 

H(x,m, Du{x))+ / -r— HB,m,Du(^))(x)mB)d^ + PB(x,t)db(t) 

Jr" 

/■ 5 

u{x,T) = h{x,m) + / —— h{^,m){x)m{^)d^ (5.32) 

dm 

dtm + {A*m — i/3^Am + div(G(x, m, Du)m))dt + PDm.db{t) = 0 (5.33) 


m{x, 0) = mo{x) 

In fact the equation for u contains two unknowns u{x,t) and B{x,t) which are both field processes adapted 
to the filtration i3*. Since the equation for u is a backward stochastic partial differential equation ( backward 
SPDE) we need the two unknowns to obtain adapted processes. 

6 MEAN FIELD GAMES 

6.1 THE PROBLEM 

In Mean Field Games, we cannot have a Bellman equation, similar to p3.8p . (13.9p . since the problem is not 
a control problem. However, for a fixed parameter m(.) we can introduce 


28 









dx = g{x, m, v{x))ds + a{x)d'w 


( 6 . 1 ) 


x{t) = X 


X 


= E[^ f{x{s),m{s),v{x{s)))ds + h{x{T),m{T))] 


( 6 . 2 ) 


and we set 


u{x,t) = inf Ja;,t(^^(-),"l(-)) 

^(•) 

in which we omit to write explicitly the dependence in m. Then u{x,t) satisfies Bellman equation 


du 

— — + Au = H{x, m, Du{x)) 
u{x, T) = /i(x, m) 


(6.3) 

(6.4) 


We next write that m must be the probability density of the optimal state, hence 


-^ + A*m + div {G{x,m, Du)m{x)) = 0 (6.5) 

m(x,0)=mo(x) (6.6) 

and this is the system of HJB-FP equations, corresponding to the classical Mean Field Games problem. 

6.2 MASTER EQUATION 

We do not have Dynamic Programming, however the uncoupling argument corresponding to the system 
(I6.3|) . (l6.4p . (l6.5|) . (16.60 can be pursued. This time, we introduce directly the Master equation by setting 


dU f d 


17(x, m, t)(Q(74*m(Q + div (G(^, m, D[/(Q)m(Q)d^ = H{x,m,DU{x)) (6.7) 


U(x, m, T) = h{x, m) 


( 6 . 8 ) 
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We then consider m{x,t) solution of the FP equation 


din 

——h A*m + div {G(x, m, DU(x, m))m{x)) = 0 
at 

m{x, 0) = mo{x) 


We next introduce 


(6.9) 


u{x,t) = U{x,m{t),t) (6.10) 

in which m{t) = m{x,t) is the solution of (j6.9|) . In combining (16. 7|) and (|6.9I) we obtain easily (j6.3|) . in 
which m{t) is the solution of (16.91) . The master equation (16. 7|) looks very similar to that of mean field type 
control, see (15.51) . (I^IUD . and even simpler, since the derivative in m of the Hamiltonian does not appear 
anymore. However, we may loose a very important symmetry property 

-^U{x,m,t){C) ( 6 . 11 ) 

6.3 MASTER EQUATION IN U 

We want to associate to the Master equation (16.71) (|6.8|) an equation in the space R. We do not have a 
Bellman equation, so we shall proceed directly with (16.7p . (|6.8p . To simplify we assume (j{x) = I, so we 
rewrite the Master equation as 

—+ yAa,[/+y J A^—U{x,m,t){0'm{0d^ + J D^—U{x,m,t){i)-G{i,m,D^U{i)))m{i)d^+ 

( 6 . 12 ) 


+H{x,m,DxU{x)) = 0 


U(x, m, T) = h{x, m) 


We next take the gradient in x to obtain 


o 2 2 /* p 

—D^U+y^xDxU+yDx j A^—[/(x,m,t)(Qm(Qd^+T>a, J D^—U{x,m,t)iO-G{C,rn,DU{C)))m{C)d^+ 

(6.13) 

+DxH{x, m, DxU(x)) + D^U(x, m, t)G{x, m, DxU) = 0 
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We then replace x hy X and we want to interpret the equation which is obtained. We set V({X,t) = 

d d 

DxU{X,Cx,t), then of course —DxU{X,Cx,t) = Next 


D^H{X,m,D,U{X)) = D^H{X, CxM{X)) 


We have next , see (15.flip 


m, + DIU{X, m, t)G{X, m, D^U{X)) = (6.14) 

DU{X,t)G{X, CxM{X)) 

There remains to interpret A^OxU{X) + X^-^U{X,m,t){^)m{^)d^. We first have 

{DU{X)T,T)= [ AxU{X,m)m{x)dx 

Dx{{DU{x)r,r) = {D^u{x)r,r) = 

DxAxU{X,£x) + Dx / A^—U{(,m,t){X)miOdC 
Jjjn om 

which is the quantity we want only if we make the symmetry assumption 


-^U{x,m,t){i) = ^U{i,m,t)[x) (6.15) 

So under the assumption (|6.15h . we claim that U{X,t) is the solution of the Master equation 

+ ^D‘^U{T,T) + DU{X)G{X,CxM{X)) + DxH{X,CxM{X)) = 0 (6.16) 


U{X,T)=Dxh{X,Cx) 


However, (I6.15p implies that DU is self-adoint. So we are looking to a solution U{X,t) such that DU{X,t) 


is a self-adjoint operator inT-L . As we shall see in the linear quadratic case, this might not be true. 


Note that , applying (|6.15p at time T we need to have, in particular 


^/i(x,m)(0 = -^h{i,m){x) 


(6.17) 


In the mean-field control problem, since U{X,T) = Dh{X,£x), we have DU{X,T) = D‘^h{X,Cx), it is 


31 










a self adjoint operator by construction. 


Therefore there remains an advantage for equation (I6.12|) . There is still a possibility to use the space 


as follows. Instead of considering the derivative DxU{x,m) one considers the map U{x,m) itself. We then 


write U{x,m) = U{x,Cx)- One then interpret directly the Master equation (|b.l2p . We have successively 



A^^U{x, m, 
am 


Dj,U{x,X,t){T,r) 


r 

/ D^—U{x, m, t)(0-G(C, m, D^U{C)))m{C)d^ 

j^n om 


{DxU{x, X, t), G{X, Cx, DxU{X, Cx,t)) 


so we get the Master equation 


dU 


2 2 

+ y + ^DlU{x, X, t){T, T) + {DxUix, X, t), G{X, Cx,DxU{X, Cx,t)) 


(6.18) 


+H{x,Cx,DxU{X,Cx,t)) = 0 


Uix,X,T) = h{X,Cx) 

7 STOCHASTIC MEAN FIELD GAMES 

7.1 THE MODEL 

We recall how to obtain a system of HJB-FP equations, see [2] for more details. 

Consider the notation of section 18.2.11 We recall that the state equation is the solution of 


dx = g{x, m{t),v{x, t))dt + a{x)dw + /3db{t) (7-1) 

x(0) = Xq 

and B^=a{b{s), s <t) , J^=a{xo, b{s),w{s), s < t). This time m{t) is a given process adapted to with 
values in We again consider feedback controls which are field processes adapted to the filtration B^. 

We follow the theory developed by Shi Ge Peng m- We define the function 

pT 

u(x,t) = inf / f{x{s),m{s),v{x{s),s))ds + h{x{T),m{T))] (7.2) 

^(•) Jt 
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and show that it satisfies a stochastic Hamilton-Jacobi-Bellman equation. The equation is 


—dtu + {Au — -(3‘^Au)dt + /3^div B{x, t)dt = if(x, m{t), Du{x))dt + (3B{x, t)db{t) (7-3) 

u{x,T) = h{x,m{T)) (7.4) 

If we can solve (17.3p . (17.41) for the pair u{x,t), B{x,t) of adapted to field processes, then a verification 
argument has been described in [2], to show that u{x,t) coincides with (j7.2p . 

Next, consider the optimal feedback v{x,m{t), Du{x,t)) and impose the fixed point property that m{t) 
is conditional probability density of the optimal state, we get the stochastic FP equation 

dtm + {A*m — + div(G(x, m, Du)m))dt + j3Dm.db{t) = 0 (7.5) 


m(x, 0) = mo(x) 

We thus have obtained the pair of HJB-FP equations for the stochastic mean held game problem, 
(j7.3|) . (l7.4|) and (17.5p . 

7.2 THE MASTER EQUATION 

We derive the Master equation by writing 


u{x,t) = U{x,m{t),t) 

The calculation has been done in [2]. We obtain 


-^ + AU- ^13^ AU + - ^/3'Am(Q + div (G(^, m, DU{0)m{C)m+ (7-6) 

+/3^div(/ ^7/(x,m,t) ^ 1 2 7 7 d U{x, m, t) rj)Dm{^)Dm{rj)dCdr] = H (x, m, DU (x)) 

Jr" om 2 dm^ 

U(x, m, T) = h{x, m) 


7.3 THE MASTER EQUATION IN THE SPACE U . 

We take the particular case a{x) = I then dZSI) becomes 
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m 


^(a2+/32)A.C/-l(a2+/32) 


A, 


dU{x, m, t) 
dm 


JR’ 


D, 


dU{x, m, t) 
dm 


{i).G{i,m,DU{0)m{i)di 

(7.7) 


+/3^div(/ f f ^ (g, r])Dm{^)Dm{r])dCdr] = H (x, m, DU (x)) 

Jr" dm 2 Jr„ Jr„ dm^ 

We assume the symmetry property (|6.15p . We take the gradient in x and obtain 

d cP' + [P (P + P r d 

—Da;U-\ - - - A^Da;U-\ - - - J A^—U{x,m,t){^)m{^)d^+ (7.8) 

+D^ / D^—U{x,m,t){i).G{i,m,DU{p))m{i)di + Z)j,id(x, m, L)a;f7(x)) + DlU{x,m,t)G{x,m, D^U)- 
Jr” dm 

-/3^div(/ {C)Dm{C)dC) + f f ^ ^^'^'^\ c,r])Dm{C)Dm{r])dCdr] = 0 

Jr” dm 2 Jk" Jr” dm^ 

So introducing Z^(X) = DxU{X, Cx) and we replace x by A in the equation (17.8p . We have interpreted 
in (I6.16P all the terms not including /3.We want to check that 


A,D,UiX)+Dj A^—UiX,m,m)miOdC- (7.9) 

Jjjn om 

-2D^diY{ [ f f ^ ^^^'^'^\ ^,r])Dm{^)Dm{r])d^dr] = D^U{X){ek,ek) 

JR” (J'lJT' JR” JR” C'm 

Now we have 


DV({X){ek, Cfc) = f AU{x,m)m{x)dx + f f -^—U{x,m){y)Dm{x).Dm{y)dxdy (7.10) 

JR” JR” JR” dm 


Next we take the derivative in m of the right hand side and then the gradient in x to obtain 


C d 

Dj;A^U{x,m) + / A^—U{^,m,t){x)m{^)d^+ 

JR” dm 


+D, 


[ [ v)Dm{^)Dm{y)d^dy-2D^diY ( f {rf^Dm{y)dy 

jR” JR” VJR" 
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We need again to use the symmetry property and replace x by X to prove (ESI). We can now write the 


Master equation in the space % , namely 


d (j'i flS 

—U + —D^U{V,T) + Y E D^l^iX){ek,ek) + DU{X)G{X,CxM{X)) + D,H{X,CxM{X)) = 0 (7.11) 

k=l 

U(X,T) = D^h(X,Lx) 

If we do not have the symmetry property, then we consider the Master equation for U{x,m,t), namely 
(EZD, and we introduce the argument X instead of m, as in (16.181) . We use (12.151) to state 



DxU{x,X,t){ek,ek) 

k=l 


Dj,Uix,X,t){T,T) 


Next 


div ( f = - E -^i^k, DxU{x, X,t)) 

JR" am oxk 

We can then write the Master equation for U{x,X,t) 


dU a 

'dt^~ 2 




a 


A,U + —Dj,U{x, X, t)T, r) + {DxU{x, X, t), G{X, Cx,D,U{X, Cx,t))+ 


(7.12) 


g2 n n 

+ Y E X>‘xU{x, X, t)(efc, Cfe) + E 

^ k=l k=l 


d 

dxk 


{ek,DxU{x,X,t)) + H{x,Cx,D^U{x,X,t)) = 0 


U{x,X,T) = h{x,Cx) 


8 LINEAR QUADRATIC PROBLEMS 

8.1 ASSUMPTIONS AND GENERAL COMMENTS 

For linear quadratic problems, we know that explicit formulas can be obtained. We have solved the Master 
equations in U(x, m, t) in our paper [2]. So we shall here solve only the Master equation in the space Ti. We 
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then take 


f{x, m, v) = -[x*Qx + v*Rv + {x — Sy)*Q{x — Sy)] 


( 8 . 1 ) 


g{x, m, v) = Ax + Ay + Bv 


( 8 . 2 ) 


h{x, m) = - [x*Qtx + {x- SxyYQTix - STy)] 


(8.3) 


in which we have noted y = ■ We also assume 


it(x) = a, hence a{x) = a = aa* 


(8.4) 


We deduce easily 


H (x, m, q) = ^x*{Q + Q) x - x*QSy + ^y*S*QSy 
-^q*BR-^B*q + q*{Ax + Ay) 


(8.5) 


G{x,m,q) = Ax + Ay — BR ^B*q 


( 8 . 6 ) 


8.2 MEAN FIELD TYPE CONTROL MASTER EQUATION 

We begin with Bellman equation (|3.23p . namely 


1 1 R 

— + -D^ViX){aiX)T,aiX)T) + -/32 ^ UV(A)(efc, e^) + ER(A, W(A)) = 0 (8.7) 

k=l 

V{X,T)=Eh{X,Cx) 


and using (15.lip we get 


Ot /’I -1 72 

_ + -D^V{X){a{X)T, a{X)T) + Y. ek)+ 

k=l 


( 8 . 8 ) 
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fE (-X*{Q + Q)X - X*QS1KX + -EX^S^Q^EX - -DV*BR-^B*DV + DV*{AX + XEX) ) = 0 

V{X, T) = ^E [X*QtX + {X- 5tEX)*Qt(^ - 5tEX)] 

We look for a solution of (|8.8p . of the form 

V{X,t) = ^EX*P(t)X + i(EX)*S(t)EX + \{t) (8.9) 

We have clearly 


P{T) = Qt + Qt, S(T) = S^QtSt - (S^Qt + QtSt), A(r) = 0 (8.10) 


Next 


DViX, t) = P{t)X + E(t) EX 
D^V{X, t)Z = P{t)Z + S(t) EX 

therefore 

ZlV(X)(c7(X)r,(T(X)r) =trP(t)a 


^ D^V{X){ek,ek) = tr {P{t) + S(t)) 

k=l 

With these calculations, we can proceed in equation (j8.8|) and obtain 


lExXpm + liEX)XmEX + ±X(t) 

+ hraP{t) + ytr(P(t) + S(t)) + 


+E 


^X*{Q + Q)X - X*QSEX + i(EX)*S’*Q5EX- 


— (P(t)X + J:{t)EX)*BR-'-B*{P{t)X + S(t)EX) + (P(t)X + S(t)EX)*(XX + XEX) 


= 0 
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We can identify terms and obtain 


+ ^traP(t) + ytr(P(t) + S(t)) = 0 


therefore, from the final condition A(T) = 0, it follows 


Mt) = [ (^traP(s) + ^tr(P(s) + S(s))ds (8.11) 

Jt ^ ^ 

Identifying quadratic terms in X( within the expected value) and in EX respectively, it follows easily that 


and 


(W 

dt 


+ PA + A*P - PBR-^B*P + Q + Q = 0, P{T) = Qt + Qt, 


( 8 . 12 ) 


dT, 

dt 


+ S(y4 + y4 - BR-'^B*P) + {A + A- BR-^B*PYY.- 


(8.13) 


-T,BR-^B*J: + S*QS -QS - S*Q + PA + A*P = 0 


S(T) = S^QtSt — (S^Qt + QtSt) 


We obtain formula (j8.1jp with the values of P(t), 5](t), \{t) given by equations (I8.12p . (I8.13p . (I8.1ip . 
We turn to the Master equation, namely (I5.25D 


Q 2 -i n 

-U + ^D^U{T, T) + Y. D^^{X){ek, ek) + DU{X)G{X, Cx,UiX))+ 


(8.14) 


k=l 


dH 


D,HiX,Cx,U{X)) +EY,uiY)D^^{Y,Cx,UiY)){X) =0 


UiX,T)=DhiX,Cx) 


We have 


G{X,Cx,ld{X)) = AX + AEX - BR-^B*U{X) 


D^H{X,CxM{X)) = {Q + Q)X -QS¥.X + A*U{X) 
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= —X 


QSi + {j r]mir])dr]yS*QSC + q*AC 


so 


^Y,uiY)D^^{Y,Cx,U{Y)){X) = {-S*Q + S*QS)EX + {A)*m{X) 


and (|8.14p reads 



k=l 


+{Q + Q)X + {-QS - S*Q + S*QS) EX + A*U{X) + {A)*m{X) = 0 


We expect U{X,t) = P{t)X + S(t)EX to be the solution. This is satisfied at time T. We check easily 
that equation (18.1511 is satisfied with the choices of P{t) and S(t) given by (18.1211 . (18.1311 . noting that 
DU{X,t)Z = P{t)Z + S(t)EZ. We see also that (I8.15p could be consider directly, without referring to 
Bellman equation (|8.8p . 

8.3 MEAN FIELD GAMES MASTER EQUATION 

This time, there is no Bellman equation. We may consider the Master equation 



k=l 


U{X,T) = D^h{X,Cx) 

which becomes in the LQ case 



k=l 


+ {Q + Q)X -QSEX + A*U{X) = 0 


U{X, T) = {Qt + Qt)X - QtStEX 


But we must have DU{X,t) self adjoint. At time T we have 
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DU{X, T)Z = {Qt + Qt)Z - QtSt^Z 


and therefore we need to assume 


QtSt = S^Qt 


(8.18) 


Next if we look for a linear solution U{X,t) = P{t)X + 'E,{t)KX, then we must have P{t) and S(t) 
symmetric. Checking in (18.170 we see that P{t) satishes the Riccati equation (18.121) . but S(t) satishes 


dt 


+ S(y4 + ^ - BR-'^B*P) + {A- BR-^B*P)*11- 


(8.19) 


-T,BR-^B*T. -QS + PA = {) 


s(r) = -QtSt 


and therefore S(t) cannot be symmetric unless 


A = Q,QS = S*Q 


( 8 . 20 ) 


If these assumptions are not satisfied the equation (I8.17p has no solution satisfying DU{X,t) self adjoint. 
In that case we may use (|7.12l) which in the LQ case reduces to 


dU (t2 + ^2 


a 


dt 


+ ^A^U+—Dj^Uix, X, t)(T, T) + {DxUix, X, t), AX+AEX-BR-^B*D^UiX, Cx,t))+ (8.21) 


q 2 n n 


{ek,DxU{x,X,t)) + l:X*{Q + Q)x 


k=l 


k=l 


dx 


x*QSEX+ 


+ ^EX*S*QSEX - ^{D^U{x,X,t)TBR-^B*D^U{x,X,t) + {D^U{x, X,t)T{AX + AEX) = 0 


U{x, X, T) = ]^x*{Qt + Qt)x - x*QtStEX + ]^EX*S^QtStEX 
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We look for a solution of the form 


U{x, X, t) = ^x*P{t)x + + ^EX*T{t)EX + /i(t) 


( 8 . 22 ) 


We must be careful that we cannot have E{t) symmetric. It is already true at time T. We have 


P(T) = Qt + Qt 

(8.23) 

S(T) = -QtSt 

(8.24) 

r{T) = S*tQtSt 

(8.25) 

KT) = 0 

(8.26) 


Next 


D^U{x, X, t) = P{t)x + S(t)EX 
DxU{x, X, t) = E*{t)x + r{t)EX 
DlU{x,X,t) = P{t), D\U{x,X,t)Z = r{t)EZ 
We use these formulas in the Master equation (18.211) . to obtain 

lx*j^P{t)x + x*jE{t)EX + i(EX)*|r(t)EX + + a^)tTP{t) 

+iE*{t)x + T{t)EXy{A + A- BR-^B*{P{t) + E{t))EX) + ^/3^trr(t) + (3hTE{t) + 

^x*{Q + Q)x - x*QSEX + i(EX)*5*Q5(EX)- 

-i(P(t)x + E{t)EX)*BR-^B*{P{t)x + E{t)EX) + {P{t)x + E{t)EX)*{Ax + AEX) = 0 
Identifying terms, we obtain 

^P{t) +PA + A*P - PBR-^B*P + Q + Q = 0 (8.27) 
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— + EM + - BR-^B*P) + (A* - PBR-^B*)'P - T.BR-^B*Y.-QS + PA = Q (8.28) 

at 

7T-\ 

+ T{A + A-BR-^B*{P + S)) + (A + i - BR-^B*{P + S))*r+ (8.29) 

+S*QS - T.BR-^B*'B + Ei + i *E = 0 

^ trP(t) + ^tr r(t) + E(t) = 0 (8.30) 

Remark 4. In our previous work [2] we have solved completely the Master equation in the LQ case, for 
mean field type control and mean field games, considering U{x,m,t) with m S L^(M’^). So m was not 
necessarily the density of a probability. Although everything can be carried out, the calculations are much 
more complex than in the current case working with C/(x, X, f).In particular we had to keep a lot of terms 
depending on mi = In the case U{x,X,t) this term is 1 and calculations simplify greatly. This 

shows the advantage of = L^(n, A, P; M”) with respect to L^(M”'). We work with a Hilbert space in both 
cases, but we keep the properties of probabilities in the case of R. 

9 NEW CONTROL PROBLEMS 

9.1 INTERPRETATION OF THE FIRST ORDER BELLMAN EQUATION 

We call first order Bellman equation, the equation 


—+ER(X,£x,UH(X)) = 0 (9.1) 

ot 

H(X,r) =Eh(X,£x) 

We associate to this equation a “deterministic” control problem in the state space R. The state equation is 
given by 


— =g{X{s),Cxi,),v{X{s),s)) (9.2) 

X{t) = X 

The control is defined by a feedback v{X, s) with values in M'^.We define the cost functional 

JxAO)= r^f{Xis),Cx(s),viXis),s))ds + Eh{XiT),CxiT)) ( 9 . 3 ) 
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and the value function 


V{X,t) = inf JxAO) (9.4) 

v{.) 

We claim that the value function satisfies (|9.ip . Note that although the equation (19.21) is deterministic, 
the trajectory is random, because the initial condition X is random. Nevertheless, because the equation is 
deterministic, we say that (19.2p . ( 19.3p is a deterministic control problem in "H. We provide a formal proof 
based on the optimality principle. We write 


V{X,t)=inf 

vi.) 


Ef{X{s),/:xis),v{X{s), s))ds + V{X{t + e),t + e) 


~ inf [eE/(W, Cx,v) + V{X + eg{X, Cx,v),t + e)] 

from which we get easily 

^ + inf [E/(X,£xw) + {DViX,t),giX,£x,v))] = 0 

but 


inf [EfiX, £x,v) + {DV{X, t),g{X, £x , u))] 


E inf [f{X, £x,v)+ DV{X, t).g{X, £x , u)] = 

V 


= EH{X,£x,DV{X,t)) 


and (19.11) follows immediately. 


9.2 INTERPRETATION OF THE FIRST STOCHASTIC BELLMAN EQUATION 


The first stochastic Bellman equation is 


— + '^D‘^V{X){N,N) +EH{X,£xM{X)) 

V{X,T)=Eh{X,£x) 


(9.5) 
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where N is standard gaussian independent of X. We associate to this equation a stochastic control problem in 
the space H . We assume that on the probability space P we can construct a standard Wiener process 
with values in M"’ w{t) which is independent of the random variable X. Considering again a feedback v{X, s) 
we define the Ito equation in the space "H 


We define the pay-off 


dX = (7(X(s), s))ds + adw{s) 

X{t) = X 


Jx,t 



W.f{X{s),Cx(s),v{X{s),s))ds + m{X{T),CxiT)) 


and the value function 


(9.6) 


(9.7) 


V{X,t) = inf Jx,t{v{.)) 

v{.) 

We claim that the value function is the solution of (IH31). We proceed formally as above . We have to 
evaluate V{X{t + €),t + e). But 

X{t -I- e) ~ X -I- eg{X,Cx,v{X)) + a{w{t -|- e) - w{t)) 

Therefore 


V{X{t + e), t + e) ~ e^{X, t) + e{DV{X, t),g{X, £x,v))+ 


+V {X + a{w{t + e) — w{t)),t) 


and 


V{X + a{w{t + e) 


w{t)),t) ~ V{X,t) + ^{D^V{X,t)N,N) 


We can then obtain (|9.5|) easily 


9.3 INTERPRETATION OF THE SECOND STOCHASTIC BELLMAN EQUATION 


By second stochastic Bellman equation we mean 
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(9.8) 


2 -t Tl 

— + ^D^ViX)iN, N) + Y. D^V{X){ek,ek) + ¥.H{X,Cx,DV{X)) = 0 

k=l 

v{x,T) = m{x,Cx) 

We consider on Vt,,A^P the variable X, the standard Wiener process w{t) as above independent of X and a 
new standard Wiener process h{t) with values in M”', independent of X and of w{t). We set = a{b{s), s < t). 
A feedback control v{X, s) on T-L is now a random field on T-L, such that, for X fixed, it is a stochastic process 
adapted to the filtration B^. We define the SDE on T-L 

dX = g{X{s), Cx(s)^v{X{s), s))ds -\- adw{s) -\- fTdh{s) (9-9) 

X{t) = X 

However, now Cx{s) does not refer to the probability law of X(s) but to the conditional probability given 
B^ . To figure out what it is, we consider the process T(s) defined by the equation 


dY = g{Y{s) + /3b{s), CY(s)+ 0 b{s),v{Y{s) /3b{s)))ds adw{s) (9.10) 

Y{t) = X - /3b{t) 

in which b{s) must be considered as a given continuous function, not a stochastic process. The only 
randomness in the model (j9.10|) is X and w{s). The equation (|9.1Up is similar to (19.Op since b{s) is not 
stochastic. The conditional law of X(s) given B^ is CY[s)+i 3 b{s) > so can write (19.9p as follows 


dX = g{X{s),CY{s)+pb{s),v{Y{s),s))ds adw{s) + /3db{s) (9.11) 

X{t) = X 

The payoff corresponding to the feedback u(W(s), s) is now defined by 

JxA^i-)) fiYis),CY(s)+i3b(s),vi^{s),s))ds+ (9.12) 

+h{X{T),CY(T)+ 0 b{T))\^^^Y{t) = X] 
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The conditioning expesses the fact that we have access to the hltration . We define the value function 


V{X,t) = inf Jx,t(u(.)) 
^(■) 


(9.13) 


Normally V{X,t) for fixed X is random and measurable. In fact, it will be independent of B*^ and 
the solution of (j9.8p . This is due to the independence of X and B^. We first write 


ViX,t)r^miE[efiX,Cx,viX)) + 

v{.) 


E 



f {s)+0b(s) 


v{X{s), s))ds + h(X(r), (r)+;3MT))l^(t + e),B^+% X{t) 



B\X{t) = X 


From the optimality principle, it follows 


F(X, t) ~ inf E[e/(X, Cx,v{X)) + E{V{X{t + e),t + e)\B^+^,X{t) = X)\B\X{t) = X] (9.14) 

v{.) 


Since 


X{t + e) ~ X + eg{X,Cx,v{X)) + a{w{t + e) - w{t)) + /3{b{t + e) - b{t)) 

when we condition with respect to we must consider b{t + e) — b{t) as hxed. Therefore we can check 

easily that 


E{V{X{t + e),t + e)\B^+^,X{t) =X) 


V{X, t) + e^{X, t) + e{DV{X),g{X, Lx , t^(X))) 


2 n 

+e—(T»V(X)iV, X) + ^Y.^DV{X), ek)[bk(t + e) - 6fc(f))+ 

k=\ 

+W ^ (T)^V"(X)efc,e/)(6fc(t + e) - bkit))(bi{t + e) - bi{t)) 

^ k,l=\ 

in which 6fc(f) represent the coordinates of b{t). Plugging this formula in (|9.14p and completing calculations 
we obtain dlSD. 
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10 THE MAXIMUM PRINCIPLE 


10.1 PRELIMINARIES 

It is natural to consider the maximum principle in the context of the new control problems considered in the 
previous section. We shall need to change slightly the notation. We set M = L?‘{Q.,A,P), hence Ti. = M". 
We should see M as replacing the space M of real numbers. We shall need M'^ for the space of controls. 
A control is a function E(s) with values in M^. When there is no Wiener process w{s) or 6(s), there is no 
aspect of adaptation. So V(s) is a “deterministic” function. Of course, this is paradoxical since the values 
of E(s) are random variables, but mathematically the values are in a Hilbert space M*^. When we introduce 
w(t) and b(t) we will say that it is a stochastic process. 

10.2 DETERMINISTIC MAXIMUM PRINCIPLE 

We consider the differential equation 


^=g(X(t),£j,^,j,v(t)) ( 10 . 1 ) 

A(0) = A 

in which the control v(t) takes values in M'^.The solution X(t) takes values in M”. The particularity of 
the space M” is that we may consider the random variable together with its probability distribution . 

We then define the payoff functional 

J(v(.))= rEf(X(t),£x(t),v(t))dt + Eh(X(T),£x(T)) (10.2) 

Jo 

The control v{t) can be defined by a feedback v{t) = v{X(t),£x(t)A)- We recall the Dynamic Program¬ 
ming approach to this problem . We have 


inf J(u(.))=U(A,0) 

d(.) 


in which the value function V {X, t) is the solution of the first order equation 


(10.3) 


dV 


+ EH{X,£x,DViX)) = 0 


V{X,T)=Eh{X,£x) 


(10.4) 
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in which we recall the notation 


H{x, m, q) = inf(/(x, m, v) + q.g{x, m, v)) 

and we have denoted by v{x,m,q) the point of minimum. Next, consider for Q in T-L = M^,the new 
Hamiltonian 


HiX,Cx,Q)= inf [Ef{X,Cx,v) + {Q,g{X,£x,v))] (10.5) 

in view of the definition of the scalar product the infimum is attained at v{X,Cx, Q)- Therefore 


H{X,Cx,Q)=^H{X,Cx,Q) 

We next define the Lagrangian 


L{X,Cx,v,Q)=¥.f{X,Cx,v) + {Q,g{X,Cx,v)) (10.6) 

in which X,Q £ M” and v £ M'^. There is a slight abuse of notation for v which is an element of in 
the usual defintion of the Hamiltonian and an element of M'^ in the definition of the Lagrangian. We have 


HiX,Cx,Q)= inf LiX,Cx,v,Q) (10.7) 

and the infimum is attained at v{X, Cx-,Q)- Bellman equation (110.41) reads 


(J V 

— + H{X,Cx,DV{X))=f) (10.8) 

V{X,T)=W.h{X,Cx) 

We note that V{X,t) = V{Cx,t) so DV(X,t) = Ha;—xw^^-^(^)- Therefore 

dm 

v{X,/:x,DV{X,t)) = v{X,Cx,D f^^^^'^\ x)) 
which is a feedback in X as defined above. We set also 
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FiX,Cx,Q) = fiX,Cx,viX,£x,Q)) 
G{X,Cx,Q) = 9 {X,Cx,v{X,Cx,Q)) 


( 10 . 9 ) 


and 


H{X,Cx,Q)=F{X,Cx,Q) + Q.G{X,Cx,Q) 

We shall need DxH{X,Cx,Q)- By that we mean that Q is a fixed element of M”" and we take the 
gradient with respect to X. Since H{X, Cx, Q) = ^H{X, Cx, Q) we can compute the gradient according to 
formula (j2.8p which means 

f) M 

DxH{X, Cx,Q) = D,H{X,£x,Q) + —(F, £x, Q){X) (10.10) 

in which F is a copy of X . We have also 


DxHiX,Cx,Q) = D,F{X,Cx,Q) + {D,Gnx,Cx,Q)Q+ (10.11) 

(X) + EygiD^^riY, £x, Q){X)Q 
om om 

Set U{X,t) = DV{X,t), then Bellman equation reads 

dV 

— + H{X,£x,U{X,t))=0 (10.12) 

Differentiating in X we obtain the Master equation 

dlA 

— + DxH{X,CxM) + DU{X)G{X,CxM) = ^ ( 10 . 13 ) 

Consider the optimal control problem (llO.ip . (IIP.2D . the optimal control u{t) is given by 

u{t) = v{X{t),Cx(t)M{X{t),t)) ( 10 . 14 ) 

and using the same notation X{t) for the optimal state we have 

g{X{t),Cx(t)Mt)) = G{X{t),Cx(t)M{X{t),t)) ( 10 . 15 ) 
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L{X{t),Cx(t),u{t),U{X{t),t)) = H{X{t),Cx(t)M{X{t),t)) 


(10.16) 


Define now Z{t) = U{X{t),t), the co state, we have 

^ + DU{Xit),t)G{X{t),Cxit),l^iX{t),t)) 

so, collecting results from (|1U.13|) to pU.lOp we obtain the following system 


dX 

dt 


g{X{t),Cx{t),u{t)) 


= DxL{X{t),Cxit),u{t),Z{t)) 


XiO) = X 


(10.17) 


Z{T) = Dxh{X{T),CxiT)) 

u{t) minimizes L{X{t), Cx{t),v^ ^it)) in n G (10.18) 

where the Lagrangian is defined in pO.Op . For completion , we express 


DxL{X{t),Cx{t),u{t),Z{t)) = D^f{X{t),Cx{t),u{t)) +EY(t)u{t)Dx-^f{Y{t),Cx{t),u{t)){X{t))+ 

+ {Dxg)*iX{t),Cx(t),uit))Z{t) +^Y{t)Z{t)u{t) (O.X/(Y{t)Xxi,yu(t))(X(t))Z(t) (10.19) 

in which, as usual Y(t) is a copy of X(t). For the sake of verification, let us compute {DxL{X{t), Cx{t)i'^{^)^ Y{t)),X(t) 
two ways. First from the definition of the Lagrangian we can write 

{DxL{X{t),Cx(t ), <t), Z{t)),X{t)) = {Dx^f{X{t),£xit), u{t)),X{t))+ 

+ iDx{g{X{t),£x(t ), n{t)), Z{t)),X{t)) 

From formula (|2.8I] we see that Dx^f{X{t),Cx(t)ju{t)) is indeed equal to the term in / in the right-hand 
side of piO.lOp . Next 
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{Dx{g{X{t),Cxit ), u{t)), Z{t)),X{t)) = {Dxg{X{t), Cx^t ), u{t))X{t), Z{t)) 


From formula (|2.1Up we can write 


Dxg{X{t),Cx{t),uit))X{t) = D^g{X{t),Cx{t),uit))X{t)+ 


+^Y{t)x{t)^y-^9{X{t), Cx{t ), u{t)){Y{t))X{t) 

therefore 


{Dxg{X{t),Cxit ), u{t))Xit), Z{t)) =eJ 2 Ziit)^{X{t),Cxit), u{t))Xj {t)+ 

ij 

+^z{t)x{t)u{t)'^ Zi{ty^Y{t)x(t)^.-^9i{X{t), Cx{t),u{t)){Y {t))Xj{t) = I + II 
i 9j 

Consider the second term, which is the only point to check.We can exchange the names of X{t) and Y{t) 
which are identical, hence 

II = ——5fi(y(t),£jY(i),M(t))(X(t))Xj(t) 

i ^ 

Now, referring to (|1U.19D and testing the formula with X{t),we obtain a formula which is identical to 
I + //. So we can state 

{DxL{X{t),£xit),uit),Z{t)),X{t)) = {DxEf{X{t),£xit),u{t)),X{t))+ (10.20) 

+{Dxg{X{t),£xit), uit))X{t), Zit)) 

Exercise 5. We can derive the system (|10.18p directly, not as a consequence of the Bellman equation. 

10.3 STOCHASTIC MAXIMUM PRINCIPLE 

We introduce the Wiener process w{t) with values in M”, independent of the initial condition X and the 
filtration U* generated by X and w{s), s < t. A control v{t) is a stochastic process adapted to U* with values 
in M'^. We then define the state of the system by the SDE in 
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dX = g{X{t), Cx{t),v{t))dt + adw{t) 
X{0) = X 


( 10 . 21 ) 


and the pay off is defined by 

J{v{.)) = rEf{Xit),Cx(t),vit))dt + Eh{XiT),CxiT)) 

Jo 

We know that 


( 10 . 22 ) 


infJ(t;(.)) = y(X,0) 


in which V (X, t) is the solution of the Bellman equation 


(10.23) 


— + y B»V(X)(iV, N) + H{X, Cx,DV{X)) = 0 (10.24) 

V{X,T)=Eh{X,Cx) 

The optimal control tt(t) is obtained by minimising the Lagrangian ( see llO.Oll L{X{t), Cx(t),v,iJ{X{t),t)) 
in u S with U{X{t),t) = DV{X{t),t). The function U{X,t) is the solution of the Master equation 

— + ^{D^UiX)N,N) + DxH{X,Cx,U)+DU{X)G{X,Cx,U) = 0 (10.25) 

Define Z{t) = U{X{t),t) , where X{t) is the optimal state ( we use the same notation), then, by Ito’s 
calculus is the space M” we have 


dZ{t) = DU{X{t),t)g{X{t),£x{t), u{t))dt + aDU{X{t),t)dw{t) + ^{D‘^U{X{t))N, N)dt + ^{X{t),t)dt 

(Jl 

therefore from (I10.25D it follows, setting K{t) =aDU{X{t),t), 


—dZ{t) = DxL{X(t), Cx{t),u{t), Z{t))dt — K{t)dw{t) (10.26) 

Z{T) = DxhiX{T),CxiT)) 
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We have also the state equation 


dX{t) = g{X{t),Cx(t)-iU{t))dt + adw{t) (10.27) 

X(0) =X 

and the optimality condition for u{t) is expressed by 

u{t) minimizes L{X{t), Cx{t):'v, Z{t)) in v ^ M‘^ (10.28) 

The triple (ll0.26D . (ll0.27p . (ll0.28l) expresses the stochastic maximum principle. The adjoint equation 

(jl0.26p is a BSDE in the space M®*. 
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